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Abstract 

Artificial spike-based computation, inspired by models of computations in the central ner- 
vous system, may present significant performance advantages over traditional methods for spe- 
cific types of large scale problems. In this paper, we study new models for two common instances 
of such computation, winner-take-all and coincidence detection. In both cases, very fast con- 
vergence is achieved independent of initial conditions, and network complexity is linear in the 
number of inputs. 



1 Introduction 

Recent research has explored the notion that artificial spike-based computation, inspired by models 
of computations in the central nervous system, may present significant advantages for specific types 
of large scale problems EI El El EH E] ■ This intuition is motivated in part by the fact that 
while neurons in the brain are enormously " slower" than silicon based elements (about six orders of 
magnitude in both elementary computation time and signal transmission speed), their performance 
in networks often compares very favorably with their artificial counterparts even when reaction 
speed is concerned. In a sense, evolution may have been forced to develop extremely efficient 
computational schemes given available hardware limitations. 

In this paper, we study new models for two common instances of such computation, winner- 
take-all and coincidence detection. In both cases, very fast convergence is achieved and network 
complexity is linear in the number of inputs. 

We first present a simple network of FitzHugh-Nagumo (FN) neurons for fast winner-take-all 
computation. In contrast to most existing studies, e.g. the recent jS], the network's initial state can 
be arbitrary, and its convergence is guaranteed in at most two spiking periods, making it particularly 
suitable to track time- varying inputs. If several neurons receive the same largest input, they all spike 
as a group. 

Using a very similar architecture, but replacing global inhibition by global excitation, we obtain 
an FN network for fast coincidence detection, in a spirit similar to Again the system's response 
is practically immediate, regardless of the number of inputs. 

In section 2 we review basic properties of the FitzHugh-Nagumo model in its dimensionless 
version. Sections 3 and 4 discuss applications to the design of fast winner-take-all networks and 
coincidence detection networks. Brief concluding remarks are offered in Section 5. 

1 To whom correspondence should be addressed. 



2 The FitzHugh-Nagumo Model 



The FitzHugh-Nagumo model |5] is a well-known simplified version of the classical Hodgkin-Huxley 
model the first mathematical model of wave propagation in squid nerve. Originally derived 
from the Van der Pol oscillator ^5], it can be generalized using a linear state transformation to the 
dimensionless system [Tl] 

v — v(a — v)(v — l)—w + I -j. 
w = j3v — 

where a,/3, 7 are positive constants. Here v models membrane potential, w accommodation and 
refractoriness, and / stimulating current. 

Simple properties of the FN model |14) can be exploited for neural computations. For appropriate 
parameter choices, there exists a unique equilibrium point for any given value of /. Furthermore, 
this equilibrium point is stable, except for a finite range /; < I < Ih where the system tends to a 
limit cycle. The steady-state value of v at the stable equilibrium point increases with /. 



3 Winner-Take- A 11 Network 

Winner-take-all (WTA) networks, which pick the largest element from a collection of inputs, are 
ubiquitous in models of neural computation, and have been used extensively in the contexts of 
competitive learning, pattern recognition, selective visual attention, and decision making ^ 1181 
I19| . Furthermore, Maass J3| showed that WTA represents a powerful computational primitive as 
compared to standard neural network models based on threshold or sigmoidal gates. 

The architectures of most existing WTA models are based on inhibitory interactive networks, 
implemented either by a global inhibitory unit or by mutual inhibitory connections. Many studies, 
such as [I], require the system dynamics to be initiated from a particular state, which prevents real- 
time tracking of time- varying inputs. Starting with many WTA implementations in analog 
VLSI circuits have been proposed. While they do guarantee a unique global minimum, dynamic 
analysis is difficult and computation resolution limited. Studies of spike-based WTA computation, 
as in 0, are comparatively recent. In this section, we describe a very simple network of FitzHugh- 
Nagumo neurons for fast winner-take-all computation, whose complexity is linear in the number of 
inputs. The network's initial state can be arbitrary, and its convergence is guaranteed in at most 
two spiking periods, making it particularly suitable to track time- varying inputs. If several neurons 
receive the same largest input, they all spike as a group of winners. 
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Figure 1: Diagram of the network. There are n FN neurons receiving different external stimulating 
inputs. A global inhibition neuron monitors the network. 
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As illustrated in Figure the network consists of n FN neurons. Each neuron receives a 
stimulating input Ii from outside as well as a common inhibition current z from the global inhibition 
neuron. The dynamics of the FN neurons [i = 1, . . . , n) are 



( 



v t . = Vi{a - Vi){vi - 1) - Wi + Ii - z 



The dynamics of the global inhibition neuron switches between a charging mode and a discharging 
mode. It starts charging if there is any FN neuron spiking in the network, i.e. when Vi exceeds a 
given threshold value vo- It switches to discharging if the state is saturated (enough close to the 
saturation value in simulation) and stays at this mode until next time a FN neuron spikes. The 
specific dynamics of these two modes can be very general. For simplicity, we use 



where z is a constant saturation value, and k c and kd are the charging rate and discharging rate. 

Within at most two periods, the winner will be the only neuron spiking in the whole network. 
In some cases, this is achieved within the first period. 

To perform WTA computation, we set the charging rate of the global neuron to be fast and the 
discharging rate to be slow. Thus, similarly to [H], if there is any FN neuron spiking, the strength of 
the inhibition current increases to its saturation value very rapidly, leaving no chance for the other 
neurons to spike. The global neuron then discharges slowly, which lets the FN neurons smoothly 
approach the oscillation region. The first neuron entering the oscillation region will be the one with 
the largest input. So it spikes as the winner and ignites a new period. Note that before enter the 
oscillation region, all FN neurons converge to their equilibrium points with the equilibrium point of 
the winner having the largest value. A slowly discharging process allows the winner to occupy the 
highest position and helps it to spike immediately once it enters the oscillation region. Given the 
parameters of the FN neurons, the frequency of the result depends on the global neuron's saturation 
value, its charging and discharging rates, and the value of the largest input. If we also fix the global 
neuron dynamics, the frequency increases with the increasing of the largest input. Simulation results 
are shown in Figure 



• Initial conditions and computation speed The mechanism described above guarantees 
that initial conditions can be set arbitrarily, which cannot be realized by most of the previous WTA 
models. With appropriate parameters, the computation can be completed at most in two periods. 
The first spiking neuron is chosen by initial conditions, while the second one is the neuron with the 
largest input, which remains the winner until the inputs change. Actually, if the initial inhibition 
is set large enough so that all the FN neurons are depressed in the beginning, then the neuron with 
the largest input will spike first. The computation speed of our FN network is faster and more 
robust than the WTA model recently presented in |5], whose network of integrate-and-fire neurons 
has to wait until the winner gets the right to spike, which may take a long time for large networks. 

• Varying inputs and noise Since initial conditions do not matter in our model, the network 
can easily track time- varying inputs. Figure [3] illustrates such an example, where three inputs 
switch winning positions several times. The spiking neuron always tracks the largest input. The 
computation is robust to signal noise as well. 

• Multiple winners Decreasing the global neuron discharging rate kd extends the waiting 
time before the winner enters the oscillation region. This is helpful if there exist several neurons 
receiving the same largest input and we expect them all spike as a group of winners. Enlarging the 
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Figure 2: Simulation result of WTA computation with n = 10. The parameters of the FN neurons 
are set as a = 5.32, (3 = 3, 7 = 0.1, with spiking threshold vq = 5 . The inputs li are chosen 
randomly from 20 to 125. The parameters of the global neuron are zq = 160, k c = 1, kd = 1/50. All 
the initial conditions are chosen arbitrarily, (a) States Vi versus time. The dashed curve represents 
the state of the neuron receiving the largest input, (b) State z versus time. 



time neurons stay in the stable region allows these neurons with the same input converge to each 
other, and to enter the oscillation region and spike simultaneously ( Figure 0J. Note that in [5], 
only one winner can succeed and it is picked arbitrarily from the group of candidates. 

If the network size is small, the network may be augmented with all-to-all couplings between 
FN neurons, with the coupling gain increasing with the similarity of the inputs (e.g., of the form 
e - «l ij—U I y This lets the neurons receiving identical inputs converge together exponentially 
(using partial contraction theory |lf>| ) and thus provides another solution to the multiple- winner 
problem. 

• Computation resolution Computation resolution can be improved by decreasing the global 
neuron discharging rate kd while increasing the charging rate k c . Decreasing kd allows the winner 
fully distinguished with the following neurons; increasing k c prevents the following neurons spike 
after the winner. Figure 0] illustrates such an example with winners I m ax = 120 while the second 
largest input I\q = 119.5. The resolution here is much better than the WTA models presented 
previously, including [1 11 117) . It can be further enhanced by decreasing the relaxation time of the 
FN neurons. 

• Input bounds The inputs to the FN neurons should be lower-bounded by Ii (the lower 
threshold of the oscillation region) to guarantee that the neurons can spike before the inhibition is 
fully released. They should also be upper bounded to set zq. 

• K- Winner- Take- All K- Winner- Take- All (fc-WTA) is a common variation of WTA computa- 
tion |13j . where the output indicates for each neuron whether its input is among the k largest. An 
example fc-WTA circuit in JJ] extends the WTA model in by formulating the problem in terms 
of mathematical programming, but it inherits its low resolution limit from jllj as well. Conversely, 
the advantages of the above FN network generalize to the fc-WTA case. As inhibition decreases, 
the FN neurons enter the oscillation region rank-ordered by their inputs. For WTA computation, 
we charge the inhibition neuron after the first arrival. For fc-WTA computation, we only need to 
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Figure 3: Simulation result of WTA computation with varying input. The parameters are all equal 
to those in Figure except that n = 3. Inputs change continuously. The winner always tracks the 
neuron with the largest input. 



modify the charging moment to capture the k arrival instead. Since the neurons enter the oscil- 
lation region in sequence, they spike in sequence. If we set k = n, we get a pre-ordered spiking 
sequence in each period, which may be used to realize soft- WTA [151 H5| . and also provides a simple 
desynchronization mechanism for binding problems 6 . The computation resolution follows directly 
from that in WTA. A detailed description and discussion of the A:- WTA network will be presented 
separately. 

• Spike-controlled coupling and slow inhibition The feedforward and recurrent connections 
used in our WTA network are similar to those in [Hj , where a "universal" control system is developed 
based on olivo-cerebellar networks. The couplings inside the circuit are also spike-controlled and 
they use a FitzHugh-Nagumo-like model containing four variables. A similar mechanism is also used 
in [IB], where WTA is implemented to compute the object with the largest size. Slowly-discharged 
inhibition is also used in biologically motivated models such as ^3 ^] . 

• Computational complexity The complexity of the network is 0(n). Since the FN neurons 
are independent, they can be added or removed from the network at any time. 



4 Fast Coincidence Detection 

Recent neuroscience research suggests that coincidence detection plays a key role in temporal bind- 
ing ^21- Hopfield et al. [2] proposed two neural network structures, both able to capture a "many- 
are-equal" moment, to model speech recognition and olfactory processing. A similar computation 
can be implemented by FN neurons, with faster and more salient response. 

Consider a leader- followers network with a structure similar to Figure [2 except that the global 
neuron (the leader) is now excitatory, and the connections from the leader to the followers are 
unidirectional. For simplicity, we assume that all the neurons are FN neurons with the same 
parameters but different inputs. The dynamics of the leader (v ,w ) obeys equations while 
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Figure 4: Simulation result of WTA computation with multiple winners. The parameters are the 
same as in FigureEJexcept that k c — 5, kd — 1/80. The inputs are I\ = ■ ■ ■ = 1% = 120, Jio = 119.5 . 
The first plot shows the Vi's as functions of time. The dashed (red) line represents v±o and the solid 
(blue) lines the other Vi's. The first nine neurons converge together during the waiting time and 
spike simultaneously as a group of winners. The second plot, an enlarged version of the first at 
a spiking moment, shows that i>io is completely depressed by the winners even though the input 
difference is very small. 



those of the followers (i — 1, . . . , n) are 



where k(vo — Vi) is the coupling force from the leader to the followers. Neurons i and j synchronize 
only if inputs h and Ij are identical. We define the system output accordingly to capture the moment 
when this condition becomes true for a large number of inputs, as illustrated in Figure[SJ Note that 
the coupling gain k should be large enough to guarantee synchronization (an explicit threshold can 
be computed analytically |16p. but not so large as to have the leader numerically dominate the 
dynamic differences between the followers. More general formal studies of synchronization can be 
found in |16) . based on nonlinear contraction theory. 



5 Concluding Remarks 

Basic computations such as winner-take-all and coincidence detection can be performed fast and 
robustly using extremely simple spike-based models. The results are currently being extended to 
higher-level perception problems. 
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Figure 5: Simulation result of fast coincidence detection with n = 30. The leader and the followers 
are FN neurons with the same parameters as in Figure El The inputs are Iq = 90 and I\ , . . . , /30 
varying from 20 to 80, and the coupling gain is k = 1.7. The upper plot shows Y^i=i max (0, Vi) 
versus time, and the lower 7i ,...,/„ . 
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